YAFIMA: Yet Another Frequent Itemset Mining Algorithm

Efficient discovery of frequent patterns from large databases is an active research area in data mining with broad applications in industry and deep implications in many areas of data mining. Although many efficient frequent-pattern mining techniques have been developed in the last decade, most of them assume relatively small databases, leaving extremely large but realistic datasets out of reach. A practical and appealing direction is to mine for closed or maximal itemsets. These are subsets of all frequent patterns but good representatives since they eliminate what is known as redundant patterns. The practicality of discovering closed or maximal itemsets comes from the relatively inexpensive process to mine them in comparison to finding all patterns. In this paper we introduce a new approach for traversing the search space to discover all frequent patterns, the closed or the maximal patterns efficiently in extremely large datasets. We present experimental results for finding all three types of patterns with very large database sizes never reported before. Our implementation tested on real and synthetic data shows that our approach outperforms similar state-of-the-art algorithms by at least one order of magnitude in terms of both execution time and memory usage, in particular when dealing with very large databases.